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To evaluate the portfolio in assessing interprofessional learning between 
medical and midwifery students. The student portfolio is assessed using a 
rubric consisting of four assessment criteria. A total of 32 student portfolios 
were tested for reliability coefficients and interrater agreements. We 
conducted an in-depth interview with mentors and focus group discussion 
(FGD) with students for exploring their perceptions of the ability of the 
portfolio to assess the learning. Interview and FGD data were converted into 
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1. INTRODUCTION 

The implementation of interprofessional education (IPE) in the health profession education is 
expanding globally. This is supported by the agreement between various health professions to develop 
consensus on interprofessional core competencies as the basis for curriculum development and 
interprofessional learning.[1] However, assessing interprofessional competences is quite challenging. [2] It 
needs robust assessment system including standardized tools, longitudinal assessment, and variety of data. 
[2—4] It also demands a system that support students to develop the knowledge, skills, attitudes and values to 
work colaboratively in an interprofessional team [5-6]. 

A portfolio is an assessment method that can describe the authenticity of learning. It also can 
encourage the student to be responsible for the learning and development of his or her own competence by 
collecting evidence of learning experience or performance [7-8]. This method has widely been used in 
various contexts of medical education, including undergraduate, professional, postgraduate and sustainable 
professional development students [8-9]. Buckley et al. [10] state that the portfolio has the effect on learning, 
i.e., able to improve the ability of the integration of theory and practice, reflection and introspection, and self- 
study. The use of a portfolio in community-based medical education can stimulate reflective learning and 
underpin the program planning and supervision process [11]. The study of portfolio use in interprofessional 
learning is still limited [3]. Domac et al. [3] concludes that student reflection on the portfolio can illustrate 
the achievement of student interprofessional competencies. 

In Indonesia, as one of the institutions that wish to prepare health workers ready for collaborative 
practice, the Faculty of Medicine of Universitas Sebelas Maret (FM-UNS) has developed a community-based 
IPE learning module for undergraduate medical and D3 midwifery students. This module is a pilot project to 
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incorporate IPE into medical education curricula (IPE FM-UNS team). It uses family health services, 
especially maternal and child health as the context of its learning with a four week learning time. The 
learning objectives have been developed based on the four core competency domains by the Interprofessional 
Collaborative Practice (IPCP) team as a guide for developing the IPE curriculum [1]. For assessing the 
students’ IPE learning, we design a portfolio with the rubric as the IPE portfolio assessment guide. 

To assess the quality of the IPE portfolio, evaluating of the scoring system with specific criteria is 
required [12-13]. So that the assessment system and the learning program may be accountable to the public 
[14]. Validity and reliability are among the evaluation criteria that aim to determine whether the assessment 
system indeed assesses the content or competence to be tested or not [15-16]. 

Interestingly, there are no significant researches that studied the use of portfolios in interprofessional 
learning of midwifery students. So far, the study of portfolio evaluation has been limited to only one health 
profession (medical or midwifery only) [11,17—21], while portfolio evaluation on interprofessional learning 
is still limited [3]. 

This study evaluated the IPE portfolio content by exploring the views of faculty lecturers as IPE 
portfolio assessors and investigating student perceptions as the subjects of IPE learning. It also reviewed the 
reliability of the IPE portfolio rubric. 


2. RESEARCH METHOD 

A mixed methods research design was done in this study, combining quantitative and qualitative 
design with the aim of gaining a broad understanding of research questions. [22] To evaluate the reliability if 
the IPE portfolio, cross-sectional survey strategy was chosen, while case study strategy was used to explore 
the views of supervisors and students. In case study strategy, a program, event, or activity was in-depth 
explored by using a variety of qualitative data collection procedure [22]. 


2.1. Sample and Setting 

For the quantitative data, the sampling method was total sampling. All results of the portfolio rubric 
were included in this study, due to the small population (n = 32, undergraduate medical students = 17, D3 
Midwifery students = 15). For the qualitative data, there were 17 IPE students (n = 17, medical students = 10, 
D3 midwifery students = 7) and all mentors/assessors (n = 4) who were the samples in this study. 


2.2. Portfolio Assessment Procedures 

A portfolio was used to assess the interprofessional learning. Undergraduate medical and D3 
midwifery students attended in the learning with the context of maternal and child health services. The 
portfolio consisted of four sections based on the learning period in week 1 to 3 and ended with the final 
reflection in week 4. During the learning, each group consists of 5-6 students were guided by two mentors, 
each of whom was a lecturer of medicine and midwifery. Each week students would do mentoring that aims 
to stimulate reflection, monitor learning, and provide feedback. At the time of assessment, each student 
portfolio will be assessed separately by two assessors who constitute one midwifery and one medical mentor 
from another group using the criteria-based assessment rubric. Both assessors had been trained to use the 
assessment rubric before assessing the students’ real IPE portfolio. The students' portfolio was assessed using 
the following four criteria of assessment: 

1. Level of reflection 

2. Use of Evidence of Learning 

3. Use of relevant References 

4. Quality of work plan 

Based on the four criteria above, the students’ portfolio was assessed objectively using a quantitative 
scale of O to 3, and the total value was 12. Portfolio assessment was done in each section so that four sub- 
values were added to the final value. At the beginning of interprofessional learning, the students have 
received an explanation on how they were assessed using the portfolio assessment rubric. 


2.3. Data Collection and Analysis 

The data collection procedure in this research consisted of three forms, one form of quantitative 
design and two forms of qualitative design. In quantitative design, portfolio assessment results were 
employed to calculate the validity and reliability of the rubric. Data analysis used the SPSS software for 
windows edition 21.0 or the data analysis. In qualitative design, an in-depth interview was done to faculty 
mentors and Focus Group Discussion (FGD) students as a method of data collection. This design is under the 
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purpose of case study strategy, which is for in-depth exploration of a program(22) which, in this study, it was 
the use of portfolio as an instrument of assessment of interprofessional competence. 

The recording of in-depth interview and FGD was then converted into verbatim transcripts followed 
by data analysis with content analysis techniques using Atlas.ti for windows software. After the data 
preparation and selection of unit analysis, the data were read, interpreted as a whole and coded. The coding 
process was done by two medical education experts using open coding technique. Both coders examined one 
transcript of the same in-depth interview mentor and one transcript of the same FGD student separately, and 
subsequently, the level of agreement between the code of analysis was calculated until it reached an adequate 
level of agreement (Kappa = 0.87). Finally, the coders analyzed the remaining transcript consecutively until 
forming code, group, category and meaning [23]. 


2.4. Ethical Consideration 

The ethical clearance to use student IPE portfolio data issued by the Ethics Committee of RSUD dr. 
Moewardi Surakarta, Central Java, Indonesia. Students and lecturers had been informed that participation in 
this study was voluntary. Also, the students were well informed that the participation would not affect their 
study grades. 


3. RESULTS AND ANALYSIS 
3.1. Characteristics of respondents 

There were 32 total portfolio scores, each of which consisted of 4 sub-scores and had calculated the 
reliability and validity of the portfolio rubric. Characteristics of Respondents shown in Table 1. 


Table 1. Characteristics of Respondents 
Profession Gender Total 
Male Female 








Students 

Medicine 5 10 15 
Midwifery - 17 17 
Tota 3 27 32 
Mentors/Assessors 

Medicine - 2 2 
Midwifery - 2 2 
Total - 4 4 





3.2. Rubric Reliability 
The data analysis was performed using SPSS for windows edition 20.0 against 32 total portfolio 
scores. The reliability coefficient of the portfolio section showed a high score (a = 0.808) (shown in table 2). 


Table 2. Reliability Coefficient 
Cronbach’s Alpha Cronbach’s Alpha on Standardized Items 
0.808 0.817 











Table 3 shows the quality of each item of assessment criteria calculated by an interrater agreement 
analysis. It was found that all criterion items were relevant to an acceptable interrater agreement, ranging 
from 0.331 (moderate agreement) to 0.731 (excellent agreement). 


Table 3. Interrater Agreement at Each Assessment Criterion 








Assessment Criteria Weighted Kappas 
Level of Reflection 0,731 
Evidence of Learning 0,574 
Work Plan 0,331 
Reference 0,339 


significance p <0,05 





Quantitative analysis results show that the assessment criteria in the portfolio rubric have adequate 
reliability, as calculated by Alpha's Cronbach reliability coefficient of 0.808. This suggests that all items of 
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assessment criteria in the portfolio section are homogeneously measuring different aspects of the same 
variable, i.e., interprofessional competencies [24-25]. The interrater agreement can be received with a range 
of moderate to high. It represents the extent to which the data collected in this study are correct 
representations of the variables measured [25]. 

Reliability is one of the attempts to standardize the portfolio, but the rigorous standardization can 
reduce the portfolio's own objectives, i.e., learning autonomy [9], [26], [20]. However, a low level of 
reliability is also a threat to the validity of the judgment itself [27]. Therefore, there is a need for balancing to 
achieve portfolio objectives, but portfolio reliability remains cultivated to the acceptable level [11], [21]. 
Inter-rater reliability is one of the critical issues that determine portfolio quality [11], [20], [28]. There are 
several strategies for achieving adequate inter-rater reliability, including the use of standardized assessment 
criteria, the use of assessment rubrics, the presence of a trained assessment committee, the number of 
assessors more than one, and an open discussion among assessors [29-30]. In this study, a portfolio 
assessment framework consists of 4 assessment criteria, training of all members of the portfolio assessment 
committee along with simulations, a two-person assessment committee to assess one student, a discussion, 
and negotiation between assessors before and during the process of determining the portfolio scores guided 
by the portfolio assessment rubric. 


3.3. Mentor/Assessors’ Perception 

All mentors and portfolio assessors (n = 4) followed in-depth interviews, consisting of 2 lecturers of 
Medicine (Md) and two lecturers of Midwifery (Mw). There were several themes related to the validity of 
portfolio content and other themes regarding the use of portfolios to assess interprofessional competencies. 
The themes are presented with supportive quotes. 
All mentors and assessors agreed that a portfolio is an assessment method capable of assessing 
interprofessional competencies: 


“I think it is able to assess interprofessional competence." (Mw1) 
“Yes, I think it is appropriate to assess interprofessional competence." (Md1) 


In this case, the mentor also admitted that the student portfolio illustrated the development of 
interprofessional competencies during the learning process: 


“Yes, it can describe how the division of roles and responsibilities is conducted. For example, in the 
first week, a student said that she did not understand at all with this activity ..., then she tried to 
learn and to know each other ....... , then she was able to divide her task and to fill each other's role 
while serving the patient”. (Mw1) 

“Yes, it reflects an increase in interprofessional competence. Thus, they write in the portfolio how 
initially in the first week they did not know the other students, and then as long as they 
communicated and when they went together to the location, they did things together, shared duties 
when meeting patients and their families, and wrote that they could already reflect how to 
collaborate there.” (Mw2) 

“The sections I to 3 have given rise to interprofessional competencies. If we see them writing from 
the process of writing reflection section I, section 2, section 3, they have shown improvement and 
(the reflection) is getting deeper”. (Md2) 


Majority of assessors did not find it difficult to make an assessment based on an existing portfolio rubric: 


“Oh no, no difficulty to assess the portfolio .... The portfolio is clear ... how to write (score) 1, 
(score) 0 .... I have no problem in determining the assessment of the student portfolio”. (Mw2) 

“In my opinion, I do not have much trouble because the guides have been given in each score in the 
rubric." (Mw1) 


In a portfolio assessment, all mentors and assessors agreed that the mentor role was critical to guide 
students in collaborative learning. In addition, a scheduled mentoring system was also required so that the 
student portfolio could be as expected: 


“If for the mentoring process, yes it is essential... We should guide the students per week to remind 
them to write a reflection in their portfolio....also we can ask whether writing reflection is difficult 
and we can also help them”. (Mw1) 

Mentors also agreed that an individual mentoring system was required to provide specific feedback: 
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“at the time of the mentoring process, when they are writing the portfolio, each student is 
expected to meet with their mentor once or twice so that we can provide individual feedback from it, 
which we can later score when we assess using the rubric (portfolio) ”. (Mw2) 


Mentors and assessors agreed that portfolio writing could stimulate student ability in library search: 


“She was able to search for more references, study the journal according to what problems she 
found." (Md1) 


In addition, the mentors considered that one of the difficulties of writing portfolio was the difference in the 
depth of the curriculum of each profession so that the quality of student reflection was different: 


teed if students in medicine, they are familiar with the scientific culture that we teach to find articles 
and write scientific papers, then it is beneficial in filling the portfolio. In contrast, if midwife 
students, I think they are not used to (reflect), so the reflection is also different (the quality)”. (Md2) 


To improve portfolio quality, the mentors and assessors perceived that direct performance-based assessments 
could be included as evidence of learning: 

“\...(by direct assessment) we can see the collaboration, regarding communication, cooperation in 
how the division of task is made ... .. it can be used as evidence of learning in the portfolio as well 


...”. (Mw2) 


In-depth interviews with mentors and assessors illustrate their views on the use of portfolios to 
assess interprofessional learning. It is an aspect of face validity and content of the interprofessional learning 
portfolio. All mentors and assessors feel that the student portfolio illustrates the collaboration and 
improvement of interprofessional competencies. In this study, mentors are the primary source of information 
related to the development or achievement of student learning because they longitudinally follow, guide, 
assess and provide feedback from the beginning to the end of learning. [20], [31] One of the lessons for 
faculty is the on-going need for timely feedback and support to understand how to reflect, especially in the 
mentoring session. This study also shows that a more intense scientific learning experience in the educational 
curriculum also affects the quality of student reflection on the portfolio. 


3.4. Student’s Perception 

A total of 17 students attended the focus group interview (n = 17, medical students (D), n = 10, 
midwifery students (W), n = 7) divided into 2 groups (G1, G2). There were two main themes related to face 
validity of the portfolio and the use of an IPE portfolio. Themes are presented with supportive quotes. 
The majority of students agreed that portfolios could be used to assess interprofessional collaborative 
learning: 


“yes it could be if for IPE and collaboration (used to assess interprofessional competencies)” . 
(W1,G1) 


In addition, the students also mentioned some of the interprofessional competencies that could be described 
in their portfolios, among others, the competencies of interprofessional communication, interprofessional 
ethics, roles and responsibilities, and teamwork: 


“In our portfolio, we also write about this collaboration, about the skills and attitudes we learn, 
which we will ultimately know that the way we behave and appreciate other professions is 
fundamental. In the portfolio, it is also written about how we can foster good communication 


between doctors and midwives. Also, the division of authority exists (in the portfolio).......”. 
(D7,G2) 


By reflecting on each part of the portfolio, the students felt compelled to do a better interprofessional 
collaboration. 


spegi maybe if writing the reflection is done smoothly at first, then we can know how well the 
implementation of collaboration later on”. (W4, G1) 
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The students also felt that writing reflection on the portfolio helped them review the problems systematically: 


“From writing that portfolio, it makes me easier to review the problem in order. So, for example, 
this issue should be filled like this in the portfolio based on the questions. So, we will write so that 
one day we will not forget to plan, and how to solve the problem”. (D3,G1) 


However, the students also recognized that writing portfolios were a tricky thing for them because it was the 
first time they did it: 


“because this is the first time, writing the portfolio is a scourge for us. It seems that writing 
reflecting on ourselves or writing down what we have done is easy, but when we want to write, it is 


usually difficult”. (W7, G2) 


The students also recognize that the main obstacle in writing the reflection and working on the 
portfolio was because the portfolio was individual and there was confusion about whether, to be honest, or to 
write something ideal. 


“... It is more to the discussion of the portfolio because I think the portfolio is a personal thing, 
everyone has their own problems, have different perspectives ”. (D8, G2) 

“.....we are confused to make this portfolio. We write the portfolio for the real score, or we are told 
to assess ourself... honestly”. (W3,G1) 


In the process of portfolio work, the students knew how they were assessed and worked according to the 
portfolio assessment section: 


“Mmhh...yes, I know. There are item points in the portfolio assessment”. (D2, G1) 
later in running one program, this is what I am not clear if it has been in accordance to the guide”. 
(W5, G2) 


As an additional part of the portfolio assessment, the students felt that they also needed performance 
assessments directly by trained instructors : 


“For me, I like the portfolio to be an IPE assessment, but it should be accompanied by assessment 
like an observation”. (W7, G2) 
“assessment is not only from portfolio but also from direct observation by the supervisor." (D1, 


How students interpret events on interprofessional learning as a trigger of reflection is one essential 
factor in determining the acceptance of the portfolio as an IPE assessment method [3]. The ability to reflect, 
assess the shortcomings and strengths of self, and analyze the roles and responsibilities of the team is an 
important factor in the health care team [7], [9], [10]. Students recognize that their portfolio describes how 
they learn to achieve interprofessional learning goals. This portfolio also encourages them to collaborate with 
other students from different health study backgrounds better. On the portfolio, students will collect authentic 
learning evidence that illustrates the attainment of their learning objectives. Therefore, it is necessary to 
explore students’ perceptions about whether the portfolio assesses the achievement of learning objectives or 
not [20], [32-33]. 

This study supports Domac et al. [3] which states that the portfolio can be used as a method of 
assessment of interprofessional learning. In addition, the portfolio also has a learning effect in which it can 
encourage students to reflect and collaborate with other professional students. 


4. CONCLUSION 

Based on this study’s result analysis, a portfolio is a valid and reliable tool for assessing 
interprofessional learning regarding its ability to describe interprofessional learning goals and achievement 
when seen by the students and to assess interprofessional learning and competencies when seen by the 
assessors. However, the number of samples in this study is relatively small, so further research is needed by 
using larger sample quantities. Although there has been evidence through qualitative study of students’ and 
mentors’ perceptions of the ability of the portfolio to assess interprofessional competencies, further analysis 
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of the contents of the portfolio itself, including student reflection and the evidence of learning needs to be 
done to prove whether the correct writing of reflection and the evidence really describes the achievement of 
interprofessional competence. 
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